Measuring the Impact of Code Dependencies on Software Architecture Recovery Techniques

Many techniques have been proposed to automatically recover software architectures from software implementations. A thorough comparison among the recovery techniques is needed to understand their effectiveness and applicability. This study improves on previous studies in two ways. First, we study the impact of leveraging more accurate symbol dependencies on the accuracy of architecture recovery techniques. In addition, we evaluate other factors of the input dependencies such as the level of granularity, the impact of virtual call resolution, global variable usage and whether using direct dependencies provides better results than using transitive dependencies. Previous studies have not extensively studied how the quality of the input might affect the quality of the output for architecture recovery techniques. Second, we study a system (Chromium) that is substantially larger (10 million lines of code) than those included in previous studies. Obtaining the ground-truth architecture of Chromium involved two years of collaboration with its developers. As part of this work we developed a new submodule-based technique to recover preliminary versions of ground-truth architectures. The other systems that we study have been examined previously. In some cases, we have updated the ground-truth architectures to newer versions, and in other cases we have corrected newly discovered inconsistencies. Our evaluation of nine variants of six state-of-the-art architecture recovery techniques on 8 types of dependencies shows that symbol dependencies generally produce architectures with higher accuracies than include dependencies. We also observed that using a higher level of granularity (i.e., module level) and direct dependencies helps generating better architectures. Despite this improvement, the overall accuracy is low for all recovery techniques. The results suggest that (1) in addition to architecture recovery techniques, the type of dependencies used as their inputs is another factor to consider for high recovery accuracy, and (2) more accurate recovery techniques are needed. Our results show that some of the studied architecture recovery techniques (ACDC, Bunch-SAHC, WCA and ARC) scale to the 10M lines-of-code range (the size of Chromium), whereas others do not

Machine Learning for Software Dependability

Dependability is an important quality of modern software but is challenging to achieve. Many software dependability techniques have been proposed to help developers improve software reliability and dependability such as defect prediction [83,96,249], bug detection [6,17, 146], program repair [51, 127, 150, 209, 261, 263], test case prioritization [152, 250], or software architecture recovery [13,42,67,111,164,240]. In this thesis, we consider how machine learning (ML) and deep learning (DL) can be used to enhanced software dependability through three examples in three different domains: automatic program repair, bug detection in electronic document readers, and software architecture recovery. In the first work, we propose a new G&V technique—CoCoNuT, which uses ensemble learning on the combination of convolutional neural networks (CNNs) and a new context-aware neural machine translation (NMT) architecture to automatically fix bugs in multiple programming languages. To better represent the context of a bug, we introduce a new context-aware NMT architecture that represents the buggy source code and its surrounding context separately. CoCoNuT uses CNNs instead of recurrent neural networks (RNNs) since CNN layers can be stacked to extract hierarchical features and better model source code at different granularity levels (e.g., statements and functions). In addition, CoCoNuTtakes advantage of the randomness in hyperparameter tuning to build multiple models that fix different bugs and combines these models using ensemble learning to fix more bugs.CoCoNuT fixes 493 bugs, including 307 bugs that are fixed by none of the 27 techniques with which we compare. In the second work, we present a study on the correctness of PDF documents and readers and propose an approach to detect and localize the source of such inconsistencies automatically. We evaluate our automatic approach on a large corpus of over 230Kdocuments using 11 popular readers and our experiments have detected 30 unique bugs in these readers and files. In the third work, we compare software architecture recovery techniques to understand their effectiveness and applicability. Specifically, we study the impact of leveraging accurate symbol dependencies on the accuracy of architecture recovery techniques. In addition, we evaluate other factors of the input dependencies such as the level of granularity and the dynamic-bindings graph construction. The results of our evaluation of nine architecture recovery techniques and their variants suggest that (1) using accurate symbol dependencies has a major influence on recovery quality, and (2) more accurate recovery techniques are needed. Our results show that some of the studied architecture recovery techniques scale to very large systems, whereas others do not

Effective Detection of Vulnerable and Malicious Browser Extensions

Unsafely coded browser extensions can compromise the security of a browser, making them attractive targets for attackers as a primary vehicle for conducting cyber-attacks. Among others, the three factors making vulnerable extensions a high-risk security threat for browsers include: i) the wide popularity of browser extensions, ii) the similarity of browser extensions with web applications, and iii) the high privilege of browser extension scripts. Furthermore, mechanisms that specifically target to mitigate browser extension-related attacks have received less attention as opposed to solutions that have been deployed for common web security problems (such as SQL injection, XSS, logic flaws, client-side vulnerabilities, drive-by-download, etc.). To address these challenges, recently some techniques have been proposed to defend extension-related attacks. These techniques mainly focus on information flow analysis to capture suspicious data flows, impose privilege restriction on API calls by malicious extensions, apply digital signatures to monitor process and memory level activities, and allow browser users to specify policies in order to restrict the operations of extensions. This article presents a model-based approach to detect vulnerable and malicious browser extensions by widening and complementing the existing techniques. We observe and utilize various common and distinguishing characteristics of benign, vulnerable, and malicious browser extensions. These characteristics are then used to build our detection models, which are based on the Hidden Markov Model constructs. The models are well trained using a set of features extracted from a number of browser extensions together with user supplied specifications. Along the course of this study, one of the main challenges we encountered was the lack of vulnerable and malicious extension samples. To address this issue, based on our previous knowledge on testing web applications and heuristics obtained from available vulnerable and malicious extensions, we have defined rules to generate training samples. The approach is implemented in a prototype tool and evaluated using a number of Mozilla Firefox extensions. Our evaluation indicated that the approach not only detects known vulnerable and malicious extensions, but also identifies previously undetected extensions with a negligible performance overhead

How Effective Are Neural Networks for Fixing Security Vulnerabilities

Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.Comment: This paper has been accepted to appear in the proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2023), and to be presented at the conference, that will be held in Seattle, USA, 17-21 July 202

Comparing Software Architecture Recovery Techniques Using Accurate Dependencies

Abstract—Many techniques have been proposed to automati-cally recover software architectures from software implementa-tions. A thorough comparison among the recovery techniques is needed to understand their effectiveness and applicability. This study improves on previous studies in two ways. First, we study the impact of leveraging more accurate symbol dependencies on the accuracy of architecture recovery techniques. Previous studies have not seriously considered how the quality of the input might affect the quality of the output for architecture recovery techniques. Second, we study a system (Chromium) that is substantially larger (9.7 million lines of code) than those included in previous studies. Obtaining the ground-truth architecture of Chromium involved two years of collaboration with its developers. As part of this work we developed a new submodule-based technique to recover preliminary versions of ground-truth architectures. The other systems that we study have been examined pre-viously. In some cases, we have updated the ground-truth architectures to newer versions, and in other cases we have corrected newly discovered inconsistencies. Our evaluation of nine variants of six state-of-the-art ar-chitecture recovery techniques shows that symbol dependencies generally produce architectures with higher accuracies than include dependencies. Despite this improvement, the overall accuracy is low for all recovery techniques. The results suggest that (1) in addition to architecture recovery techniques, the accuracy of dependencies used as their inputs is another factor to consider for high recovery accuracy, and (2) more accurate recovery techniques are needed. Our results show that some of the studied architecture recovery techniques scale to the 10M lines-of-code range (the size of Chromium), whereas others do not. I